Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add qtpdf and qtpng exporters #1611

Merged
merged 19 commits into from
Jul 30, 2022
Merged

Add qtpdf and qtpng exporters #1611

merged 19 commits into from
Jul 30, 2022

Conversation

davidbrochart
Copy link
Member

Similar to the webpdf exporter, but using pyqtwebengine.

@davidbrochart davidbrochart marked this pull request as draft August 9, 2021 11:02
@maartenbreddels
Copy link
Collaborator

Nice work David, could you comment on the size of Qt vs Chromium, like what a minimal environment would take up in disk space?

@davidbrochart
Copy link
Member Author

@maartenbreddels the PyQt5 package takes 357M on my machine and the pyppeteer Chromium download takes 300M.

@bollwyvl
Copy link
Contributor

bollwyvl commented Aug 9, 2021

Excellent stuff!

I had played with this some on nbconvert-pdfqt (hasn't been touched in a while). This uses JupyterLab as the renderer, and indeed, I would see that as eventually becoming a desirable feature. There were some features added (e.g. --expose-app-in-browser) that would make this even easier today.

the PyQt5 package takes 357M on my machine and the pyppeteer Chromium download takes 300M

In some package managers (e.g. conda) the underlying package will be hardlinked between equivalent installations. And it would be shared with other Qt apps (matplotlib, spyder) for a given environment. While this seems like an equivalent feature to spraying stuff all over my $HOME, I'd much rather know these space restrictions at environment setup time.

But the big win, for my use case, is being able to specify/lock a single set of dependencies, with a single package manager, and get something that works without a "surprise" runtime download.

It also means it's possible to create full, platform-specific distributions (a la miniforge, etc) that contain everything needed to function on a "normal" computer: headless/docker usually missing some x11 stuff, but that list is getting shorter... and the chromium-based solutions also share this burden.

Even if QtWebEngine lags behind whatever frankenchromium comes out of some of these custom things, it's likely the Qt one is going to be packaged more robustly, and supported for longer.

Finally, pyppeteer still has some dangling issues as a hangover its multiple forks:

  • turns off SSL verification
  • seems to still have memory issues, and sometimes fails silently

self.size = self.page().contentsSize().toSize()
self.resize(self.size)
# Wait for resize
QtCore.QTimer.singleShot(1000, self.take_screenshot)
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here we wait for one second for the resize to take effect. I know it's not great but I couldn't find another way.

@davidbrochart davidbrochart changed the title Add qtpdf exporter Add qtpdf and qtpng exporters Aug 10, 2021
@davidbrochart davidbrochart marked this pull request as ready for review August 10, 2021 08:42
page_layout = QPageLayout(page_size, QPageLayout.Portrait, QtCore.QMarginsF())
else:
factor = 0.75
page_size = QPageSize(QtCore.QSizeF(self.size.width() * factor, self.size.height() * factor), QPageSize.Point)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

self.page().contentsSize() already returns a QSizeF object while QSizeF.toSize returns a QSize object.

We should be able to get away with

page_size = QPageSize(self.page().contentsSize(), QPageSize.Point)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

then we should be able to drop the line self.size = self.page().contentsSize().toSize().

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hum, actually, looking at the unit QPageSize.Point, it looks a bit arbitrary. I think we should determine that based on the DPI ratio of the "device".

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

page_size = QPageSize(self.page().contentsSize(), QPageSize.Point)

I had already tried that, but it generates a PDF with a larger size than the required page size.

Hum, actually, looking at the unit QPageSize.Point, it looks a bit arbitrary. I think we should determine that based on the DPI ratio of the "device".

I agree, I've been through these threads, but I couldn't find any relevant information about the DPI ratio.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have been doing some more experiments with this and I must say that I am very confused. I am enclined to merge the pull request as it is.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cmaureir I have spent some time today trying to figure this one out. I really think that the behavior of PyQtWebEngine here is buggy. Would love to talk about this in greater depth to our @qt friends.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As long as it's a general Qt issue, and not PyQt specific I can try to get your question to the people working in QtPdf 👍 but if it's PyQt specific, you will need to reach the folks at Riverbank.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it is likely to be Qt. I will post something a bit more specific.

Copy link
Member

@SylvainCorlay SylvainCorlay Jan 26, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cmaureir the bug is that printToPdf adds whitespace at the end of the document. It does not do so when exporting to PDF. (This only occurs when we set pagination to False or course). Multiplicating the dimensions by 0.75 magically makes all whitespace disapear.

Screenshot from 2022-01-26 14-10-08

else:
raise RuntimeError(f"Export file extension not supported: {output_file}")
self.show()
self.app.exec()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When trying your PR locally, the process gets stuck on this line.

This happens when doing:

jupyter nbconvert --to qtpdf Note.ipynb

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Weird, it doesn't happen on my side. Could you post the installed versions of your environment?

Copy link
Member

@martinRenou martinRenou Dec 6, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

``` # packages in environment at /home/martin/miniconda3/envs/nbconvert: # # Name Version Build Channel _libgcc_mutex 0.1 conda_forge conda-forge _openmp_mutex 4.5 1_gnu conda-forge alsa-lib 1.2.3 h516909a_0 conda-forge argon2-cffi 21.1.0 pypi_0 pypi async_generator 1.10 py_0 conda-forge attrs 21.2.0 pyhd8ed1ab_0 conda-forge backcall 0.2.0 pyh9f0ad1d_0 conda-forge backports 1.0 py_2 conda-forge backports.functools_lru_cache 1.6.4 pyhd8ed1ab_0 conda-forge binutils 2.36.1 hdd6e379_2 conda-forge binutils_impl_linux-64 2.36.1 h193b22a_2 conda-forge binutils_linux-64 2.36 hf3e587d_2 conda-forge bleach 4.1.0 pyhd8ed1ab_0 conda-forge brotlipy 0.7.0 pypi_0 pypi bzip2 1.0.8 h7f98852_4 conda-forge c-ares 1.18.1 h7f98852_0 conda-forge c-compiler 1.3.0 h7f98852_0 conda-forge ca-certificates 2021.10.8 ha878542_0 conda-forge certifi 2021.10.8 pypi_0 pypi cffi 1.15.0 pypi_0 pypi charset-normalizer 2.0.8 pyhd8ed1ab_0 conda-forge cmake 3.21.3 h8897547_0 conda-forge cmarkgfm 0.6.0 pypi_0 pypi colorama 0.4.4 pyh9f0ad1d_0 conda-forge compilers 1.3.0 ha770c72_0 conda-forge cryptography 36.0.0 pypi_0 pypi cxx-compiler 1.3.0 h4bd325d_0 conda-forge dbus 1.13.6 h48d8840_2 conda-forge debugpy 1.5.1 pypi_0 pypi decorator 5.1.0 pyhd8ed1ab_0 conda-forge defusedxml 0.7.1 pyhd8ed1ab_0 conda-forge deprecation 2.1.0 pyh9f0ad1d_0 conda-forge docutils 0.18.1 pypi_0 pypi entrypoints 0.3 pyhd8ed1ab_1003 conda-forge expat 2.4.1 h9c3ff4c_0 conda-forge flake8 4.0.1 pyhd8ed1ab_0 conda-forge fontconfig 2.13.1 hba837de_1005 conda-forge fortran-compiler 1.3.0 h1990efc_0 conda-forge freetype 2.10.4 h0708190_1 conda-forge gcc 9.4.0 h192d537_2 conda-forge gcc_impl_linux-64 9.4.0 h03d3576_11 conda-forge gcc_linux-64 9.4.0 h391b98a_2 conda-forge gettext 0.19.8.1 h73d1719_1008 conda-forge gfortran 9.4.0 h2018a41_2 conda-forge gfortran_impl_linux-64 9.4.0 h0003116_11 conda-forge gfortran_linux-64 9.4.0 hf0ab688_2 conda-forge glib 2.70.1 h780b84a_0 conda-forge glib-tools 2.70.1 h780b84a_0 conda-forge gst-plugins-base 1.18.5 hf529b03_2 conda-forge gstreamer 1.18.5 h9f60fe5_2 conda-forge gxx 9.4.0 h192d537_2 conda-forge gxx_impl_linux-64 9.4.0 h03d3576_11 conda-forge gxx_linux-64 9.4.0 h0316aca_2 conda-forge icu 68.2 h9c3ff4c_0 conda-forge idna 3.1 pyhd3deb0d_0 conda-forge importlib-metadata 4.8.2 pypi_0 pypi importlib_metadata 4.8.2 hd8ed1ab_0 conda-forge importlib_resources 5.4.0 pyhd8ed1ab_0 conda-forge ipykernel 6.6.0 pypi_0 pypi ipython 7.30.1 pypi_0 pypi ipython-genutils 0.2.0 pypi_0 pypi ipython_genutils 0.2.0 py_1 conda-forge jedi 0.17.2 pypi_0 pypi jeepney 0.7.1 pyhd8ed1ab_0 conda-forge jinja2 3.0.3 pyhd8ed1ab_0 conda-forge jpeg 9d h36c2ea0_0 conda-forge jsonschema 4.2.1 pyhd8ed1ab_0 conda-forge jupyter-core 4.9.1 pypi_0 pypi jupyter-packaging 0.11.1 pyhd8ed1ab_0 conda-forge jupyter_client 7.1.0 pyhd8ed1ab_0 conda-forge jupyter_core 4.9.1 py310hff52083_1 conda-forge jupyterlab_pygments 0.1.2 pyh9f0ad1d_0 conda-forge kernel-headers_linux-64 2.6.32 he073ed8_15 conda-forge keyring 23.4.0 pypi_0 pypi krb5 1.19.2 hcc1bbae_3 conda-forge ld_impl_linux-64 2.36.1 hea4e1c9_2 conda-forge libclang 11.1.0 default_ha53f305_1 conda-forge libcurl 7.80.0 h2574ce0_0 conda-forge libedit 3.1.20191231 he28a2e2_2 conda-forge libev 4.33 h516909a_1 conda-forge libevent 2.1.10 h9b69904_4 conda-forge libffi 3.4.2 h7f98852_5 conda-forge libgcc-devel_linux-64 9.4.0 hd854feb_11 conda-forge libgcc-ng 11.2.0 h1d223b6_11 conda-forge libgfortran5 11.2.0 h5c6108e_11 conda-forge libglib 2.70.1 h174f98d_0 conda-forge libgomp 11.2.0 h1d223b6_11 conda-forge libiconv 1.16 h516909a_0 conda-forge libllvm11 11.1.0 hf817b99_2 conda-forge libnghttp2 1.43.0 h812cca2_1 conda-forge libnsl 2.0.0 h7f98852_0 conda-forge libogg 1.3.4 h7f98852_1 conda-forge libopus 1.3.1 h7f98852_1 conda-forge libpng 1.6.37 h21135ba_2 conda-forge libpq 13.5 hd57d9b9_0 conda-forge libsanitizer 9.4.0 h79bfe98_11 conda-forge libsodium 1.0.18 h36c2ea0_1 conda-forge libssh2 1.10.0 ha56f1ee_2 conda-forge libstdcxx-devel_linux-64 9.4.0 hd854feb_11 conda-forge libstdcxx-ng 11.2.0 he4da1e4_11 conda-forge libuuid 2.32.1 h7f98852_1000 conda-forge libuv 1.42.0 h7f98852_0 conda-forge libvorbis 1.3.7 h9c3ff4c_0 conda-forge libxcb 1.13 h7f98852_1004 conda-forge libxkbcommon 1.0.3 he3ba5ed_0 conda-forge libxml2 2.9.12 h72842e0_0 conda-forge libzlib 1.2.11 h36c2ea0_1013 conda-forge lz4-c 1.9.3 h9c3ff4c_1 conda-forge markupsafe 2.0.1 pypi_0 pypi matplotlib-inline 0.1.3 pyhd8ed1ab_0 conda-forge mccabe 0.6.1 py_1 conda-forge mistune 0.8.4 pypi_0 pypi mysql-common 8.0.27 ha770c72_1 conda-forge mysql-libs 8.0.27 hfa10184_1 conda-forge nbclient 0.5.9 pyhd8ed1ab_0 conda-forge nbconvert 6.1.1.dev0 dev_0 nbformat 5.1.3 pyhd8ed1ab_0 conda-forge ncurses 6.2 h58526e2_4 conda-forge nest-asyncio 1.5.4 pyhd8ed1ab_0 conda-forge nodejs 16.12.0 h92b4a50_0 conda-forge notebook 6.4.6 pyha770c72_0 conda-forge nspr 4.32 h9c3ff4c_1 conda-forge nss 3.73 hb5efdd6_0 conda-forge openssl 1.1.1l h7f98852_0 conda-forge packaging 21.3 pyhd8ed1ab_0 conda-forge pandoc 2.16.2 h7f98852_0 conda-forge pandocfilters 1.5.0 pyhd8ed1ab_0 conda-forge parso 0.7.1 pyh9f0ad1d_0 conda-forge pcre 8.45 h9c3ff4c_0 conda-forge pexpect 4.8.0 pyh9f0ad1d_2 conda-forge pickleshare 0.7.5 py_1003 conda-forge pip 21.3.1 pyhd8ed1ab_0 conda-forge pkginfo 1.8.1 pyhd8ed1ab_0 conda-forge prometheus_client 0.12.0 pyhd8ed1ab_0 conda-forge prompt-toolkit 3.0.22 pyha770c72_0 conda-forge pthread-stubs 0.4 h36c2ea0_1001 conda-forge ptyprocess 0.7.0 pyhd3deb0d_0 conda-forge pycodestyle 2.8.0 pyhd8ed1ab_0 conda-forge pycparser 2.21 pyhd8ed1ab_0 conda-forge pyflakes 2.4.0 pyhd8ed1ab_0 conda-forge pygments 2.10.0 pyhd8ed1ab_0 conda-forge pyopenssl 21.0.0 pyhd8ed1ab_0 conda-forge pyparsing 3.0.6 pyhd8ed1ab_0 conda-forge pyqt-impl 5.12.3 py310h1f8e252_8 conda-forge pyqt5 5.12.3 pypi_0 pypi pyqt5-sip 4.19.18 pypi_0 pypi pyqtwebengine 5.12.1 pypi_0 pypi pyrsistent 0.18.0 pypi_0 pypi pysocks 1.7.1 pypi_0 pypi python 3.10.0 h62f1059_3_cpython conda-forge python-dateutil 2.8.2 pyhd8ed1ab_0 conda-forge python_abi 3.10 2_cp310 conda-forge pyzmq 22.3.0 pypi_0 pypi qt 5.12.9 hda022c4_4 conda-forge readline 8.1 h46c0cb4_0 conda-forge readme_renderer 27.0 pyh9f0ad1d_0 conda-forge requests 2.26.0 pyhd8ed1ab_1 conda-forge requests-toolbelt 0.9.1 py_0 conda-forge rfc3986 1.5.0 pyhd8ed1ab_0 conda-forge rhash 1.4.1 h7f98852_0 conda-forge secretstorage 3.3.1 pypi_0 pypi send2trash 1.8.0 pyhd8ed1ab_0 conda-forge setuptools 59.4.0 pypi_0 pypi six 1.16.0 pyh6c4a22f_0 conda-forge sqlite 3.37.0 h9cd32fc_0 conda-forge sysroot_linux-64 2.12 he073ed8_15 conda-forge terminado 0.12.1 pypi_0 pypi testpath 0.5.0 pyhd8ed1ab_0 conda-forge tk 8.6.11 h27826a3_1 conda-forge tomlkit 0.7.2 pyha770c72_1 conda-forge tornado 6.1 pypi_0 pypi tqdm 4.62.3 pyhd8ed1ab_0 conda-forge traitlets 5.1.1 pyhd8ed1ab_0 conda-forge twine 3.7.0 pyhd8ed1ab_0 conda-forge typing 3.10.0.0 pyhd8ed1ab_0 conda-forge tzdata 2021e he74cb21_0 conda-forge urllib3 1.26.7 pyhd8ed1ab_0 conda-forge wcwidth 0.2.5 pyh9f0ad1d_2 conda-forge webencodings 0.5.1 py_1 conda-forge wheel 0.37.0 pyhd8ed1ab_1 conda-forge xorg-libxau 1.0.9 h7f98852_0 conda-forge xorg-libxdmcp 1.1.3 h7f98852_0 conda-forge xz 5.2.5 h516909a_1 conda-forge yarn 1.22.17 ha770c72_0 conda-forge zeromq 4.3.4 h9c3ff4c_1 conda-forge zipp 3.6.0 pyhd8ed1ab_0 conda-forge zlib 1.2.11 h36c2ea0_1013 conda-forge zstd 1.5.0 ha95c52a_0 conda-forge ```

Copy link
Member Author

@davidbrochart davidbrochart Dec 6, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You have the exact same versions of the qt-related packages as I have in my local environment, so I'm not sure what is going on here.
Can you export your notebook to e.g. HTML?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry for the late answer. I can easily export to HTML though WebPDF also doesn't work, the process gets also stuck.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suspect the WebPDF issue (and the issue I'm having with the QtScreenshot) is related to the Chromium version installed on my machine. It seems to work on my other machine.

Copy link
Member

@martinRenou martinRenou Feb 9, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I could fix the chromium version issue I mentioned for the webpdf export.

Though your PR still doesn't work for me. It looks like the loadFinished event is never triggered, although the loadProgress reached 100. If I change your PR to use loadProgress instead of loadFinished (and waiting for it to reach a 100), I get an empty file for the pdf though.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you can ignore my comments then. I reproduce this issue with basic qtwebengine examples, it's not due to your PR.

setup.py Outdated
@@ -244,6 +244,12 @@ def get_data_files():
'webpdf': [
pyppeteer_req
],
'qtpdf': [
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As with pyppeteer, this likely needs to be added to test.

Another approach, as we've done on ipython, is to move all of these to setup.cfg and use the barely-documented string replacement technique: https://github.com/ipython/ipython/blob/master/setup.cfg#L78

@martinRenou
Copy link
Member

I'd be curious to know if something similar to this: #1718 would need to or could be done. Does PyQt allow passing a printBackground option? Does it provide a way to emulate the "screen" media?

@martinRenou
Copy link
Member

I'd be curious to know if something similar to this: #1718 would need to or could be done. Does PyQt allow passing a printBackground option? Does it provide a way to emulate the "screen" media?

Answering my own question. It looks like this PR works like a charm with the latest --theme fixes and --embed-images features without having to do anything :)

@davidbrochart
Copy link
Member Author

I can see that installing pyqtwebengine with pip makes the qtpdf and qtpng exporters either hang or segfault. We have seen in the past that the wheels for OSX were broken but it seems to be the case on all platforms now.
When installing pyqtwebengine through conda-forge, everything works locally. But I couldn't use a conda environment in the CI, I think it is because of our jupyterlab/maintainer-tools/.github/actions/base-setup@v1 action.

@blink1073
Copy link
Contributor

You should still be able to use conda-forge, see here.

@davidbrochart
Copy link
Member Author

Thanks @blink1073, let's try that.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

8 participants